Content providing program, content providing method, and content providing apparatus

ABSTRACT

A content providing program causing a computer to execute content provision processing including, acquiring a first viewing situation of a first viewer viewing a provided content in association with time when the content is provided; registering, on the basis of the first viewing situation, as feedback desirable content, missed content in a time segment in which the first viewer is likely to have failed to view the content; estimating, on the basis of the first viewing situation of the first viewer, a state in which the first viewer is enabled to view the content; and providing the first viewer with the feedback desirable content in a time segment in which estimation is made that the first viewer is in a viewable state.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of theprior Japanese Patent Application No. 2013-110023, filed on May 24,2013, the entire contents of which are incorporated herein by reference.

FIELD

The present invention relates to a content providing program, a contentproviding method, and a content providing apparatus.

BACKGROUND

When a speech, a lecture, or the like is addressed to a large audiencein a lecture hall or a classroom, conventionally a speaker or lecturerprovides the content of a speech, a lecture, or the like, directly tothe audience—viewers of the content. In such a case, the speaker or thelecturer is able to flexibly change the content while checking a degreeof fatigue of the audience, and the audience is able to listen to thespeech, the lecture, or the like seated without effectively beinginterrupted from the outside during a session of the above.

On the other hand, in recent years, more and more providers and audienceof content at remote places enjoy the benefit of, for example,E-learning or on-line video delivery of lecture content. In this case,since the provider and the audience of the content are at differentplaces, the content provider may be unable to recognize states of theaudience. More specifically, it is difficult for the content provider,for example, to flexibly adjust the content through checking whether theaudience is surely receiving the content, i.e., through checking, forexample, whether there is an individual who dozes and fails to obtain animportant point or whether the lecture is audible to even the audiencein the back of a lecture room.

On the audience side, concentration on the content may be disturbedwhen, for example, there is a telephone call when the content isreproduced whilst being seated. It is difficult to adjust the content toinduce the viewers to concentrate on a lecture and not to miss importantpoints of the content in accordance with the degree of the contentimportance.

Systems for monitoring situations of audience in respective scenes ofcontent delivery have been conventionally proposed. For example,Japanese Patent Application Laid-Open No. 2005-56205 and Japanese PatentApplication Laid-Open No. 2007-82022 propose such systems.

SUMMARY

When entertainment content such as a movie is provided by delivery,viewing situations of users are recognized according to how muchemotions of viewers are stimulated, the content is evaluated accordingto emotional viewing situations of the users, and a content providingservice is established on the basis of the evaluation.

However, when content of E-learning or a speech are provided bydelivery, since emotional audience reactions are not often seen inindividual expressions, it is difficult to change a content providingservice on the basis of audience emotional situations. Further, in thecase of the provision of the content of E-learning or a speech, thecontent includes important details to be communicated to the audience.Therefore, it is desired to provide the content while ensuring that theaudience does not miss the important points.

To allow the audience to view the content without missing the importantpoints of the content, the following two points are at issue. (1) Sinceaudience emotional reactions are not often seen in E-learning or alecture, reactions of other users may not be useful references toestimate when important points are voiced. (2) When the audiencereceives the content of E-learning or a lecture whilst being seated,there may be an individual who is disturbed by an incoming telephonecall or the like. Such an individual fails to acquire important content.

One aspect of the embodiment is a non-transitory computer readablestorage medium storing a content providing program causing a computer toexecute content provision processing including:

acquiring a first viewing situation of a first viewer viewing a providedcontent in association with time when the content is provided;

registering, on the basis of the first viewing situation, as feedbackdesirable content, missed content in a time segment in which the firstviewer is likely to have failed to view the content;

estimating, on the basis of the first viewing situation of the firstviewer, a state in which the first viewer is enabled to view thecontent; and

providing the first viewer with the feedback desirable content in a timesegment in which estimation is made that the first viewer is in aviewable state.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a configuration diagram of the content providing apparatus.

FIG. 2 is a flowchart of first content provision processing in thisembodiment.

FIG. 3 is a flowchart of second content provision processing in thisembodiment.

FIG. 4 is a flowchart of third content provision processing in thisembodiment.

FIG. 5 is a flowchart of the processing steps S2, S12, and S22.

FIG. 6 is a flowchart of the processing step S3.

FIG. 7 is a diagram depicting, as a specific example, an example inwhich content importance of the content C1 of the user A is evaluatedfor each of time slots.

FIG. 8 is a flowchart of the feedback desirable content registering stepS14-1.

FIG. 9 is a diagram for explaining the feedback desirable contentregistration processing.

FIG. 10 is a flowchart of the composite content creating step S23.

FIG. 11 is a diagram for explaining a composite content creating step.

FIG. 12 is a flowchart of the feedback desirable content provisionprocessing step S24-2.

FIG. 13 is a diagram for explaining the feedback desirable contentprovision processing step.

FIG. 14 is a configuration diagram of the server-side informationprocessing apparatus (the server Ts) in this embodiment.

FIG. 15 is a diagram depicting an example of the attentive audiencesensing data.

FIG. 16 is a diagram depicting an example of a window coordinate historyand a displayed content history in a screen.

FIG. 17 is a diagram depicting an example of content of one ofregistered data of the server Ts.

FIG. 18 is a diagram depicting a user actual visual attention estimationmap and window coordinate data.

FIG. 19 is a flowchart of calculation of a viewing score and a viewingreceptive level.

FIG. 20 is a diagram depicting an example of a viewing score, which isone of evaluation data.

FIG. 21 is a diagram depicting an example of a viewing situation, whichis one of the evaluation data.

FIG. 22 is a diagram depicting an example of control data of the viewingcontrollable level, which is one of the registered data.

FIG. 23 is a diagram depicting an example of importance evaluation dataof content.

FIG. 24 is a diagram depicting a registration example of feedbackcontent.

FIG. 25 is a flowchart of the content display pre-processing S2310.

FIG. 26 is a diagram depicting an example of feedback reservation data.

FIG. 27 is a flowchart of the processing step S2408.

DESCRIPTION OF EMBODIMENTS

[Overview of an Embodiment]

First, an overview of processing by a content providing apparatus and acontent providing program in an embodiment is explained.

FIG. 1 is a configuration diagram of the content providing apparatus.The content providing apparatus is a server-side information processingapparatus Ts that provides a client-side information processingapparatus Tc with content. The server-side information processingapparatus Ts (hereinafter simply referred to as server Ts) is, forexample, a personal computer or a server. The server Ts includes a CPU20, which is a processor, a memory 21, a communication device 22, a userinterface 24 such as a keyboard or a display device, and a storagemedium 23 configured to store a content providing program and the likeon the server side.

On the other hand, the client-side information processing apparatus Tcis, for example, a cellular phone, a smart phone, a pad terminal, or apersonal computer owned by a viewer of content. The client-sideinformation providing apparatus Tc includes a CPU 10, which is aprocessor, an output device 11 such as a display device configured todisplay content or a sound output device, an information collectingdevice 12 configured to sense attentive audience sensing data of theviewer, a storage medium 13 configured to store a client-side program, amemory 14, an input device 15, and a communication device 16.

The server Ts transmits content to the client-side informationprocessing apparatus Tc (hereinafter simply referred to as client Tc)via a communication line. The client Tc provides the viewer with thecontent from the output device 11. The client Tc collects, with theinformation collecting device 12, attentive audience sensing data of theviewer during the content provision and transmits the attentive audiencesensing data to the server Ts via the communication line. Theinformation collecting device 12 is, for example, a camera or a recorderthat collects, for example, information concerning a state of the viewersuch as information concerning a visual attention of the viewer on adisplay screen or information concerning whether the viewer is presentin front of the display screen, whether the viewer faces a directionother than the display screen, whether the viewer has left the front ofthe display screen because of an interrupting telephone call or avisitor, or whether the viewer has returned to the front of the displayscreen.

The server Ts processes the collected attentive audience sensing dataand calculates or evaluates a viewing situation of the viewer for thecontent. The server Ts registers, as feedback desirable content, contentin a time segment in which it is estimated on the basis of the viewingsituation that the viewer is likely to have failed to view the content.Thereafter, the server Ts causes the output device 11 of the client Tcto provide the viewer with the feedback desirable content in a timesegment in which it is estimated on the basis of a same viewingsituation of the same viewer during the same or different contentprovision that the viewer is in a viewable state.

Further, the server Ts evaluates, on the basis of viewing situations ofthe viewers, content in a time segment, in which a plurality of viewersis likely to have viewed at own wills of the viewers, to be important.The server Ts narrows down the feedback desirable content to the contentevaluated as being important and providing the content C1 via the screenof the output device 11.

In FIG. 1, the server Ts and the client Tc may be one informationprocessing apparatus. A plurality of the clients Tc may be connectableto the server Ts via the communication line. The server Ts may performprovision of content, collection of actual visual sensing data, andprovision of feedback content for each of the plurality of clients Tc.

The client Tc provides content corresponding to a user, who is a viewer.The attentive audience sensing data collected by the client Tc is storedon a system of the client Tc or the server Ts in association with theuser. That is, an identity of a certain user is confirmed by the clientTc or the server Ts. Data collection and content provision are performedfor the confirmed user.

Similarly, the content presented by the client Tc are provided accordingto a display of the output device 11. The attentive audience sensingdata collected by the information collecting device 12 of the client Tcis stored on the system of the client Tc or the server Ts in associationwith the display. That is, an identity of a screen of a display of acertain output device 11 is confirmed by the client Tc or the server Ts.Data collection and content provision are performed for the confirmeddisplay.

[First Content Provision Processing]

FIG. 2 is a flowchart of first content provision processing in thisembodiment. The processor 20 of the server Ts executes a contentprovision processing program in the program storage medium 23 to performthe content provision processing. Processing steps S10 to S24 of theflowchart are explained below.

Processing Step S10

In FIG. 1, a viewer D is seated in front of the screen of the outputdevice 11 of the client Tc. Content C1 is provided to the viewer D. Thatis, the server Ts is providing the content C1 by transmitting thecontent C1 to the client Tc of the viewer D and providing the content C1via the screen of the output device 11 (S10).

Processing Step S12: Viewing Situation Acquiring Step

While providing the viewer D with the content C1 via the screen of theoutput device 11 of the client Tc of the viewer D as explained above,the server Ts receives attentive audience sensing data for measurementof quantity of an information contact with the content C1 of the viewerD collected by the information collecting device 12 of the client Tc andacquires, that is, calculates and records a viewing situation of theviewer D for the content C1 (S12). As explained in detail below, theattentive audience sensing data includes visual attention area data of aregion to which a visual attention of a user, who is the viewer D, isdirected, event recognition data of an event that occurs in the usersuch as leaving from a seat of the user, and property data includeslogin/logoff, turning on/off of a screen saver function. The server Tsacquires a viewing situation of the viewer D for the content C1 on thebasis of the attentive audience sensing data. A calculation method isexplained in detail below.

In this embodiment, the viewing situation includes a viewing receptivelevel and a viewing controllable level. The viewing receptive level isan index indicating to which degree a viewer can view content. Theviewing controllable level is an index indicating to which degree aviewer can control a viewing situation.

The viewing receptive level is measured on the basis of the visualattention area data according to a degree of a visual attention stayingin a display region of the content. The viewing controllable level ismeasured on the basis of the event recognition data and the propertydata according to a degree that the viewer can control whether theviewer views the content without leaving the seat and without startingthe screen saver. Therefore, the viewing receptive level and the viewingcontrollable level are also explained in detail below.

Processing Step S14: Feedback Desirable Content Registering Step

The server Ts registers, on the basis of a viewing situation for thecontent C1 in the screen of the viewer D, as feedback desirable contentFX, content in a time segment in which the viewer D is likely to havefailed to view the content C1. The time segment in which the viewer D islikely to have failed to view the content C1 is a time segment in whichthe viewing receptive level of the viewing situation is low, forexample, time in which the visual attention stays in a region in thescreen where content is displayed is short.

Content in a time segment in which, in addition to the condition thatthe viewing receptive level is low, a condition that a viewer is likelyto be unable to control a viewing situation, that is, the visualcontrollable level of the viewing situation is low is satisfied may beregistered as the feedback desirable content FX.

Processing Step S20

Subsequently, the server Ts is providing the viewer D with content C1 orC2 via the screen of the client Tc of the viewer D (S20). The content C1is the same as the content C1 provided in the processing steps S10, S12,and S14. The content C2 is different content.

Processing Step S22: Viewing Situation Acquiring Step

While providing the viewer D with the content C1 or C2 via the screen ofthe output device 11 of the client Tc of the viewer D as explainedabove, the server Ts receives attentive audience sensing data formeasurement of quantity of an information contact with the content C1 orC2 of the viewer D collected by the information collecting device 12 ofthe client Tc and acquires, that is, calculates and records a viewingsituation of the viewer for the content C1 or C2 (S22). The viewingsituation is the same as the viewing situation acquire in the processingstep S12 and includes a viewing receptive level and a viewingcontrollable level.

Processing Step S24: Feedback Desirable Content Provision ProcessingStep

The server Ts provides the viewer D with the registered feedbackdesirable content FX of the content C1 via the screen of the viewer D ina time segment in which it is estimated on the basis of a viewingsituation for the content C1 or C2 in the screen of the client Tc of theviewer D that the viewer D is in a screen viewable situation (S24). Thefeedback desirable content FX is provided, for example, on anotherwindow beside a window on which the content C1 or C2 being viewed isdisplayed. As the feedback desirable content FX, content itself to befed back may be provided, a abridged version of the content may beprovided, or a screen for informing the content may be provided.

In the first content provision processing, while providing the viewer Dwith the content C1, the server Ts registers, on the basis of theviewing situation calculated from the attentive audience sensing data,as the feedback desirable content FX, the content in the time segment inwhich the viewer D is likely to have failed to view the content.Thereafter, while providing the same viewer D with the same content C1or the different content C2, the server Ts provides the viewer D withthe feedback desirable content FX targeting the time segment in which itis estimated on the basis of the viewing situation calculated in thesame manner that the viewer D is in the viewable state. Therefore, evenin content for which an emotional viewing situation is less easilyobtained such as a lecture or E-learning, by using the attentiveaudience sensing data for quantity of information contact measurement,it is possible to detect content in a time segment in which the viewer Dis likely to have failed to view the content rather than attentiveaudience sensing data for measurement for quantity of emotion for thecontent. Further, by using the attentive audience sensing data forquantity of information contact measurement, it is possible to providethe feedback desirable content in the time segment in which it isestimated that the viewer D is in the viewable state.

[Second Content Provision Processing]

FIG. 3 is a flowchart of second content provision processing in thisembodiment. The processor 20 of the server Ts executes the contentprovision processing program in the program storage medium 23 to performthe content provision processing. Processing steps S1 to S3 of theflowchart are processing steps for evaluating the importance of thecontent C1 (content importance). Processing steps S10 to S24-1 aresimilar the processing steps S10 to S24 of the first content provisionprocessing in FIG. 2. However, the processor 20 performs the processingsteps S14-1 and S24-1 taking into account the content importance. Theseprocessing steps are explained below.

Processing Step S1

In FIG. 1, a plurality of viewers including the viewer A arerespectively seated in front of screens of the output devices 11 of theclients Tc of the viewers including the viewer A. The content C1 isprovided to the viewers including the viewer A. That is, the server Tsis providing the content C1 by transmitting the content C1 to theclients Tc of the viewers including the viewer A and causing the clientsTc to output the content C1 to the screens of the output devices 11(S1).

Processing Step S2: Viewing Situation Acquiring Step

While providing the viewers including the viewer A with the content C1via the screens of the output devices 11 of the clients Tc of theviewers including the viewer A as explained above, the server Tsreceives attentive audience sensing data for measurement of quantity ofan information contact with the content C1 of the viewers including theviewer A collected by the information collecting devices 12 of theclients Tc and acquires, that is, calculates and records viewingsituations of the viewers including the viewer A for the content C1(S2). As in the processing step S12, the calculated viewing situationincludes a viewing receptive level and a viewing controllable level.

Processing Step S3: Content Evaluating Step

The server Ts evaluates, on the basis of the acquired viewing situationsof the viewers including the viewer A, content in a time segment, inwhich the viewers including the viewer A are likely to have viewed thecontent at own wills of the viewers, to have high viewing importance(S3). That is, the server Ts totalizes calculated and recordedrespective viewing situation data of the viewers including the viewer Aand calculates, on the basis of viewing situations of a plurality ofviewers who viewed time segments of the content C1, viewing importanceof the time segments. The viewing importance means the importance of thecontent (content importance). The server Ts may calculate the contentimportance from the viewing importance taking into account provisionimportance of the content designated by the provider of the content asimportant. The calculation of the content importance is specificallyexplained below.

Determination of viewing importance for each of the users is performedby, for example, evaluating that content in a time segment in which theviewing controllable level is high and the viewing receptive level ishigh has high viewing importance and evaluating that content in a timesegment in which, although the viewing controllable level is high, theviewing receptive level is low has low viewing importance. That is, timewhen the viewing receptive level is high and the viewer directs a visualattention to the content C1 on the screen at an own will of the viewerwhen the viewing controllable level is high and the viewer is in aviewable state can be regarded as a time segment in which importantcontent is provided. Conversely, time when, although the viewingcontrollable level is high and the viewer is under the viewable state,the viewing receptive level is low and the viewer does not direct thevisual attention to the content C1 on the screen at an own will of theviewer can be regarded as a time segment in which unimportant content isprovided.

When the viewers including the viewer A have a plurality of attributes,it is desirable to totalize viewing situation data of the viewers of therespective attributes and perform evaluation of content importance forthe viewers having the same attributes. For example, when the viewersincluding the viewer A are persons belonging to a sales team and personsbelonging to a development team, evaluation of content importance isperformed on the basis of viewing situation data of the viewersbelonging to the sales team. The evaluation of the content importance isused for determination of feedback desirable content for the otherviewers belonging to the same sales team. Evaluation of contentimportance based on viewing situation data of the viewers belonging tothe development team is used for determination of feedback desirablecontent for the other viewers belonging to the same development team.

The content importance for each of the time segments of the content C1is taken into account, for example, when the feedback desirable contentis narrowed down in a processing step S14-1 explained below and isfurther taken into account when timing for providing the feedbackdesirable contents is detected in a processing step S24-1 explainedbelow.

The server Ts executes the processing steps S1 to S3 on the content C2other than the content C1 in the same manner and calculates contentimportance for the content C2 as well. The content importance of thecontent C2 is taken into account, for example, when a time segment inwhich the feedback desirable content of the content C1 is providedduring the provision of the content C2 is determined in a processingstep S24-1 explained below.

Processing Step S10

In FIG. 1, the viewer D is seated in front of the screen of the outputdevice 11 of the client Tc. The content C1 is provided to the viewer D.That is, the server Ts is providing the content C1 by transmitting thecontent C1 to the client Tc of the viewer D and causing the client Tc tooutput the content C1 to the screen of the output device 11 (S10). Theviewer D is a viewer having the same attribute as the viewers includingthe viewer A. This is because the server Ts performs, on the basis ofthe content importance evaluated on the basis of the viewing situationdata from the viewers including the viewer A, the determination on thefeedback desirable content registration processing step S14-1 and thefeedback processing step S24-1 for the feedback desirable content whileproviding the viewer D having the same attribute as the viewer A withthe content C1. The content C1 is the same as the content C1 of thecontent evaluation target in the processing steps S1 to S3.

Processing Step S12: Viewing Situation Acquiring Step

As in FIG. 2, while providing the viewer D with the content C1 via thescreen of the output device 11 of the client Tc of the viewer D, theserver Ts receives attentive audience sensing data for measurement ofquantity of an information contact with the content C1 of the viewer Dcollected by the information collecting device 12 of the client Tc andacquires, that is, calculates and records a viewing situation of theviewer D for the content C1 (S12).

Processing Step S14-1: Feedback Desirable Content Registering Step

In the second content provision processing, content in a time segment inwhich the viewer D is likely to have failed to view the content andcontent importance is high is registered as the feedback desirablecontent FX on the basis of a viewing situation. That is, in thedetermination of the feedback desirable content FX, the content isnarrowed down taking into account content importance. Contents in a timesegment in which the viewer D is likely to have failed to view thecontents (a time segment in which the viewing receptive level is low)among the contents in the time segment in which the content importanceis high are determined as the feedback desirable contents FX.

Processing Step S20

Subsequently, the server Ts is providing the viewer D with the contentC1 or C2 via the screen of the client Tc of the viewer D (S20). Theprocessing step S20 is the same as the processing step S20 in FIG. 2.

Processing Step S22: Viewing Situation Acquiring Step

While providing the viewer D with the content C1 or C2 via the screen ofthe output device 11 of the client Tc of the viewer D as explainedabove, the server Ts receives attentive audience sensing data formeasurement of quantity of an information contact with the content C1 orC2 of the viewer D collected by the information collecting device 12 ofthe client Tc and acquires, that is, calculates and records a viewingsituation of the viewer for the content C1 or C2 (S22). The processingstep S22 is the same as the processing step S22 in FIG. 2.

Processing Step S24-1: Feedback Desirable Content Provision ProcessingStep

The server Ts provides the viewer D with the registered feedbackdesirable content FX of the content C1 via the screen of the viewer D ina time segment in which it is estimated on the basis of a viewingsituation for the content C1 or C2 in the screen of the client Tc of theviewer D that the viewer D is in a screen viewable situation and inwhich content importance of the content C1 or C2 being provided is low(S24-1).

Unlike the processing step S24 of the first content provisionprocessing, the server Ts provides the feedback desirable content FX inthe time segment in which it is estimated on the basis of the viewingsituation for the content C1 or C2 being provided that the viewer D isin the screen viewable situation and, in addition, the contentimportance of the content C1 or C2 being provided is low. Consequently,in the time segment in which the viewer D is in the viewable state andthe importance of content being provided is low, it is highly likelythat the viewer D views the feedback desirable content FX. Therefore, itis possible to effectively perform feedback of important content thatthe viewer D failed to view.

As explained above, according to the second content provisionprocessing, content importance is evaluated for each of the timesegments concerning the content C1 or C2 on the basis of the viewingsituations of the viewers including the viewer A in advance. Therefore,in the determination of feedback desirable content, it is possible todetermine a time segment in which content importance is high as afeedback target and appropriately extract feedback desirable content. Inthe determination concerning whether the feedback desirable content isto be provided, it is possible to determine a time segment in whichcontent importance is low as timing when the feedback desirable contentis to be provided and appropriately detect feedback timing.

[Third Content Provision Processing]

FIG. 4 is a flowchart of third content provision processing in thisembodiment. The processor 20 of the server Ts executes the contentprovision processing program in the program storage medium 23 to performthe content provision processing. Processing steps S1 to S3 of theflowchart are the same as the processing steps S1 to S3 of the secondcontent provision processing in FIG. 3 and are processing steps forevaluating the importance of the content C1 (content importance).Processing steps S10 to S24-2 are similar to the processing steps S10 toS24-1 of the second content provision processing in FIG. 3. However,composite content creation processing S23 for combining the feedbackdesirable content FX with the content C1 or C2 being provided andcreating composite content is added. The added composite contentcreation processing S23 is explained below.

Processing Step S23: Composite Content Creating Step

In this processing step, the server Ts is providing the viewer D withthe content C1 or C2 (S20) and is receiving attentive audience sensingdata and calculating and recording a viewing situation of the viewer Dfor the content C1 or C2 (S22). Therefore, when a time segment in whichit is estimated on the basis of the viewing situation that the viewer Dis in the viewable state comes, in a time segment in which contentimportance of the content C1 or C2 being provided is low after the timesegment, the server Ts combines the registered feedback desirablecontent FX of the content C1 with the content C1 or C2 being provided(S23). As in the processing step S24 in FIG. 2 and the processing stepS24-1 in FIG. 3, the time segment in which it is estimated that theviewer D is in the viewable state is a time segment in which the viewingreceptive level is high and the viewing controllable level is also high.With such a viewable state as a trigger, in a time segment in whichcontent importance of the content C1 or C2 being provided is low afterthe trigger, the server Ts combines the feedback desirable content Fx ofthe content C1 with the content C1 or C2 being provided.

Processing Step S24-2: Feedback Desirable Content Provision ProcessingStep

Thereafter, as in the processing step S24-1 in FIG. 3, in the processingstep S24-2, the server Ts provides the composite content at appropriatetiming to provide the feedback desirable content FX.

Therefore, the server Ts detects, on the basis of the viewing situation,timing for starting the creation of the composite content and createsthe composite content with the timing as a trigger. Therefore, theserver Ts only has to provide the composite content in the processingstep S24-1.

[Specific Example]

A specific example is explained below concerning the second contentprovision processing. First, the server Ts provides viewers includingthe viewer A, who are viewing content in client terminals of theviewers' seats in offices of bases, with moving image content C1 ofE-learning in a company. The server Ts acquires and records attentiveaudience sensing data of the viewers including the viewer A, calculatesviewing situations (viewing receptive levels and viewing controllablelevels) of the viewers including the viewer A from the data, and recordsthe viewing situations (S2).

From viewing situations of one hundred viewers including the viewer A inthe first day, the server Ts determined that a time segment in whichboth of the viewing receptive levels and the viewing controllable levelswere a H level at a high rate was a time segment in which viewingimportance of the moving image content A was high. The viewingimportance is the same as content importance in principle.

For a viewer D in the second day, when the viewer D was viewing themoving image content C 1in the viewer's seat, a time segment in whichboth of a viewing receptive level and a viewing controllable level of aviewing situation of the viewer D changed to a low level because of anincoming telephone call or the like was a time segment in which contentimportance of provided content was high. Therefore, the server Tsregistered the time segment as a time segment of feedback desirablecontent of the moving image content C1.

Thereafter, when the viewer D hang up the telephone, wore a headphoneagain, and turned to a screen, as the viewing situation of the viewer D,both of the viewing receptive level and the viewing controllable levelchanged to the H level. Further, at the same time, a scene of the movingimage content A changed to a time segment in which content importancewas the L level. Therefore, the server Ts provided the viewer D with thefeedback desirable content or a abridged version of the feedbackdesirable content and notified the viewer D that the viewer D was likelyto fail to view important content.

The first content provision processing is different from the secondcontent provision processing in that calculation of content importance(S3) is not performed and determination that takes into account thecontent importance is not performed.

The third content provision processing is different from the secondcontent provision processing in that the processing step S23 forcreating composite content in advance is additionally performed.

[Configuration of the Server-Side Information Processing Apparatus]

FIG. 14 is a configuration diagram of the server-side informationprocessing apparatus (the server Ts) in this embodiment. The server Tsincludes, in addition to the communication device 22 and the userinterface 24 depicted in FIG. 1, an input-data pre-processing unit 30for received attentive audience sensing data or the like and an externalcooperation unit 31 configured to cooperate with an external contentserver. Further, the server Ts stores, in the storage medium 23,computer programs for performing attentive audience sensing datamanagement processing 34, viewing situation acquisition processing 35,content evaluation processing 36, user viewing situation confirmationprocessing 37, feedback desirable content registration processing 38,feedback desirable content provision processing 39, composite contentcreation processing 40, UI or content display processing 32, andregistered data management processing 33 and executes the computerprograms to perform the processing corresponding thereto.

The server Ts has stored, in the storage medium 23 or the memory 21,accumulated data 41, generated data 42, recognition and evaluation data43, and registered data 45. The registered data 45 includes control dataof a viewing controllable level (see FIG. 22), a feedback type (notdepicted in the figures) indicating feedback form of feedback desirablecontent, designation of an importance label to be used (see FIG. 23),content (see FIG. 17), and a user type (not depicted in the figures)indicating a type of a viewer.

The accumulated data 41 retained by the server Ts includes attentiveaudience sensing data (see FIG. 15) and a displayed content history (seeFIG. 16). The generated data 42 includes feedback part distinguishingdata (see FIG. 24), feedback reservation data (see FIG. 26), andfeedback content data (not depicted in the figures). The evaluation data43 includes data of a viewing score (see FIG. 20), viewing situationdata (see FIG. 21), importance of content (see FIG. 23), and viewingimportance of content (see FIG. 23).

The server Ts executes a computer program of content provisionprocessing referring to the data explained above and generates data. Thecontent provision processing is explained in detail below.

[Processing Steps of the Content Provision Processing]

Processing steps of the content provision processing are explained. Asexplained above, the third content provision processing includes theprocessing steps same as or similar to all the processing steps of thefirst and second content provision processing. Therefore, the processingsteps S2, S3, S14-1, S23, and S24-2 of the third content provisionprocessing are specifically explained in detail.

[Processing Steps S2, S12, and S22: Viewing Situation Acquiring Steps]

FIG. 5 is a flowchart of the processing steps S2, S12, and S22. Theprocessing step S2 is viewing situation acquisition processing same asthe processing steps S12 and S22, although a target viewer or contentare sometimes different.

In the processing step S2, when providing the viewers including theviewer A with the content C1, the server Ts receives attentive audiencesensing data from the client Ts and acquires, that is, calculates andrecords viewing situations of the viewers including the viewer A for thecontent C1. The server Ts executes, as the processing step S2,processing explained below.

S201: Attentive Audience Sensing Data Acquiring Step

When providing the viewers including the viewer A with the content C1,the server Ts receives attentive audience sensing data from the clientTs and records the attentive audience sensing data (S201).

The attentive audience sensing data is collected by the informationcollecting device 12 of the client Ts. In the attentive audience sensingdata, sensing data (a facial expression of a user, a surroundingenvironmental noise, etc.) used for measuring a degree of an emotion ofa viewer is accumulated as attentive audience sensing data formeasurement for quantity of emotion. On the other hand, sensing dataused for measuring a degree of information contact with content beingviewed by the viewer such as a status of seating and leaving seat of theviewer, a visual attention area of the viewer, and terminal operation isaccumulated as attentive audience sensing data for quantity ofinformation contact measurement. In this embodiment, the attentiveaudience sensing data for quantity of information contact measurement isused. This is because, in a lecture and E-learning, it is difficult toacquire data for measurement for quantity of emotion.

FIG. 15 is a diagram depicting an example of the attentive audiencesensing data. In FIG. 15, the attentive audience sensing data includesvisual attention area data, event recognition data, and property data.

First, the visual attention area data is obtained by, for example, eyegaze tracking of a user, who is a viewer. The eye gaze tracking can beperformed by using a special apparatus that measures a motion of theeyeballs. It is also possible to simply specify a eye gaze direction ofthe user using a Web camera set on a screen B of a display device forcontent. For example, there is a technique described in, for example,Stylianos Asteriadis et al., “Estimation of behavioral user state basedon eye gaze and head pose application in an e-learning environment”,Multimed Tools Appl, 2009.

In these eye gaze detection techniques, in data of a visual attentiondirected to content, time in which the visual attention is directed tothe content is represented as data called heat map represented by thearea of a circle surrounding positions to which the visual attention isdirected per unit time and intensity of overlapping of the circle. Inthis embodiment, a circle centering on the center of a plurality ofpositions to which the visual attention is directed in the unit time andcovering the positions to which the visual attention is directed isrecorded as visual attention information in the unit time.

In FIG. 18, an example of a user actual visual attention estimation mapbetween time t0 to time t1 is depicted. As depicted in FIG. 15, in theuser actual visual attention estimation map recorded at time t1, circleinformation recorded after the last recording t0 is recorded in a file.If the unit time of the visual attention recording is included fivetimes between time t0 and time t1, five pieces of the circle informationare recorded at the maximum.

Second, the event recognition data is an output result of video analysissoftware for a video recorded by the Web camera set on the screen B ofthe display device for content and data received from software forrecording seating and leaving from seat. When a video in which a statein front of the screen B of a user A, who is a viewer, is recorded isanalyzed, from the position of the face of the person and the directionof the eye gaze of the person, it is possible to calculate in whichdirection the user A faces. When another user appears behind the user Aand the user A turns to the direction of the other user, the user A isconsidered to be uninterested in content displayed on the screen B. Suchan event for increasing and reducing quantity of an information contactwith content is recorded as an event recognition result.

In FIG. 15, an event in which the user A faces the direction of theother user and an event in which the user A is away from the seat arerecorded as an event recognition result.

Third, the property data is data in which a change in a system propertyrelated to a display information amount of the screen B of the displaydevice for content is recorded. For example, if a screen saver isstarted during content viewing, the user A may be unable to view thecontent during a screen saver operation period. A property change forincreasing and reducing quantity of an information contact with contentin this way is recorded together with a property value after change. Thechange in the property may be a change automatically performed by thesystem or may include a change voluntarily performed by the user A.

In FIG. 15, a property after change indicating that the screen saver isstarted is recorded.

S202: Displayed Content History Registering Step

The server Ts registers a displayed content history in a screen that theviewers including the viewer A are viewing (S202).

FIG. 16 is a diagram depicting an example of a window coordinate historyand a displayed content history in a screen. FIG. 17 is a diagramdepicting an example of content of one of registered data of the serverTs. FIG. 18 is a diagram depicting a user actual visual attentionestimation map and window coordinate data. In FIG. 18, an example of anactual visual attention estimation map and a window coordinate from timet0 to time t1 is depicted.

The window coordinate history and the displayed content history in theviewing screen depicted in FIG. 16 are explained. A user is sometimessimultaneously viewing a plurality of screens. A user sometimes displaysa plurality of windows in one screen. Therefore, a history is recordedto distinguish which users are viewing which windows in which screens.In what kinds of state the windows are arranged on the screens and whichcontent is displayed in the windows are accumulated as a history. Inthis case, even when content is displayed in a first window, in somecase, the first window is hidden under another second window and theuser A is not actually viewing the content in the first window.Therefore, in a table of window coordinates, not only data representingdisplay positions and display sizes of windows but also overlaying orderof the windows is recorded.

In the displayed content history, it is recorded which users displaywhich windows on which screens. In the example depicted in FIG. 16, itis recorded that window ID:1 and window ID:2 are simultaneouslydisplayed at time t0 and only window ID:1 is displayed at time t1.

In FIG. 17, concerning content, which is registered data, a URI in whichthe content is stored and provision importance in each of time segments(time slots) of the contents is recorded for each of content IDs (C1,C2, etc.). The content IDs are cited in the displayed content history.

The provision importance of content is data indicating importance howmuch a content providing side desires to inform a viewer. The provisionimportance of content is provided from a content provider.

S203: Associating Step

The server Ts associates the attentive audience sensing data in FIG. 15and the displayed content history in FIG. 16 using timestamp (S203).

S204: Viewing Receptive Level Calculating Step

The server Ts calculates, from the visual attention area data, a viewingscore for the content C1 being displayed on a window in a screen thatthe viewers including the viewer A are viewing and calculates a viewingreceptive level in a viewing situation on the basis of the viewing score(S204). The viewing score is a numerical value representing how muchamount of content is viewed by the user A.

In FIG. 18, an example of a user actual visual attention estimation mapand window coordinate data from time t0 to time t1 is depicted. In theexample, in the viewing score, a score indicating to which degree theuser A has viewed the content C1 using the visual sense is calculatedaccording to a visual attention staying area of the user A. However, ascore indicating to which degree the user A has listened to the contentC1 using the auditory sense may be calculated by taking into account thesound volume of the display screen B for content and a surrounding noiselevel.

A user actual visual attention estimation map 30 of the screen B fromtime t0 to time t1 records five circles. As explained above, the circlesrespectively indicate regions where the visual attention stayed in therespective five sections between time t0 and time t1. This means that,as the circles overlay more, time of the stay of the visual attention islonger.

Two windows are displayed on a screen B31 at time t0. One window isdisplayed on a screen B32 at time t1. Based on the premise that it isnot recorded at which timing from time t0 to time t1 the window ID:2 hasdisappeared, it may be impossible to accurately calculate by whichamounts the content C1 and the content C2 displayed on the window ID:1and the window ID:2 are respectively viewed. In this embodiment, on theassumption that an interval between time t0 and time t1 is very small, aviewing score at the point of time t1 is calculated only for the contentC1 displayed on the window ID:1 present at time t1.

A composite image 33 is an image obtained by windows of screens andcircles of the actual visual attention estimation map at time t0 andtime t1.

FIG. 19 is a flowchart of calculation of a viewing score and a viewingreceptive level. A viewing score for the content C1 displayed on thewindow ID:1 is calculated by, as indicated in step S42, distributing atotal area S_(ES) of circles located in a black frame 34 depicted in thecomposite image 33 in FIG. 18 at a ratio of the area of the window ID:1,on which the content C1 is displayed, to the area of the black frame 34.

That is, the viewing score for the content C1 displayed on the windowID:1 is calculated by multiplying the total area S_(ES) of the circleslocated in the black frame 34 with a ratio (S_(1W)/S_(DW)) of an areaS_(1W) of the widow ID:1 at time t1 to an area S_(DW) of a rectangle 34bounded by outer peripheral edges of all windows displayed between timet0 and time t1 (S40, S41, and S42).

FIG. 20 is a diagram depicting an example of a viewing score, which isone of evaluation data. In FIG. 20, [8000] is recorded as a viewingscore for the content C1 in the screen B of the user A in a time segmentfrom time t0 to time t1.

As explained above, in this embodiment, a viewing situation isrepresented by two levels, i.e., a viewing receptive level and a viewingcontrollable level. The viewing receptive level indicates to whichdegree content can be viewed. The viewing controllable level indicatesto which degree a viewing person can control the viewing situation. Boththe levels may be represented by a continuous quantity or may berepresented by a discrete quantity. In this embodiment, the levels arerepresented by levels in three stages (Low, Mid, and High).

For example, in a state in which the viewing receptive level is high, insome case, a visual attention stays in a window on which content isdisplayed and the window spreads to an entire viewing screen. In a statein which the viewing controllable level is low, in some case, controlrelated to content viewing is difficult while he/she leaves his/herseat.

In FIG. 19, evaluation of the viewing receptive level is performed usingthe viewing score calculated in step S42. In this example, the viewingreceptive level in the screen B of the user A from time t0 to time t1 isevaluated with reference to a degree of spread of the viewing score (akind of area) of a window displayed at time t1 into a screen area. In anexample depicted in FIG. 19, when a viewing score SCORE concerning awindow ID:Y exceeds a certain fixed value (¼) with respect to a screenarea S_(SCRN), the viewing receptive level is evaluated as High (YES inS44, S45). When the viewing score SCORE exceeds the next fixed value(⅛), the viewing receptive level is evaluated as Middle (NO in S44,S46). When the viewing score SCORE is smaller than both of the fixedvalue (¼) and the fixed value (⅛), the viewing receptive level isevaluated as Low (NO in S43, S47). As the fixed values, assuming that aneighboring value of a maximum score of the viewing score is the area ofthe viewing screen, when an amount exceeding ¼ of the maximum score is aviewing score of the window ID:Y, the viewing receptive level is assumedto be a High level. At time t1, when a plurality of windows are present,a maximum viewing receptive level in the windows can be calculated as aviewing receptive level of the entire screen.

FIG. 21 is a diagram depicting an example of a viewing situation, whichis one of the evaluation data. In FIG. 21, as the viewing situation forthe content C1 in the screen B of the user A in the time segment fromtime t0 to time t1, it is recorded that the viewing receptive level isMiddle and the viewing controllable level is Middle.

S205: Viewing Controllable Level Calculating Step

Referring back to FIG. 5, the server Ts calculates the viewingcontrollable level, which is the other of the viewing situation, on thebasis of the event recognition data and the property data of theattentive audience sensing data (S205).

FIG. 22 is a diagram depicting an example of control data of the viewingcontrollable level, which is one of the registered data. In thisexample, Facing to other people, Away their seat, Facing to Screen, Backto their seat, Screen ON: False, Screen Saver OFF: False, Login ON:True, and the like are registered.

The viewing controllable level is changed on the basis of a property,which is an event for incrementing or decrementing the registeredviewing controllable level. For example, as depicted in FIG. 22, aregistered event “Facing to other people” is an event in which theviewing controllable level is to be changed to Low because a user facesanother user. A property change “Login ON: True” indicating that a loginstate is changed to ON is a property change in which the viewingcontrollable level is to be changed to High.

An event and a property change that occur between time t0 and time t1are accumulated as the attentive audience sensing data depicted in FIG.15. When an event or a property change that affects the viewingcontrollable level is present in the attentive audience sensing data,the event or the property change is associated with events andproperties depicted in FIG. 22. A control value for the event registeredin FIG. 22 in advance is adopted as a viewing controllable level of thetime segment.

In the example depicted in FIG. 22, since a control flag is “absolutevalue change”, when an event or a property change occurs, a value of theviewing controllable level is immediately set as a control value for theevent or the property change. On the other hand, for example, when thereare a plurality of events or property changes within a fixed time,control values for the events or the property changes may be added to orsubtracted from the present level and changed as relative values.

When an absolute value change is performed, when there are a pluralityof events or property changes within a fixed time, a control value maybe determined preferentially for a last event or property change withina fixed time or a most strict or loosest control value may be adopted.In the example of the evaluation data of the viewing situation depictedin FIG. 21, a viewing receptive level and a viewing controllable levelin a certain screen of a certain user in a certain time segment areassociated and recorded. When needed, viewing situations may beevaluated and recorded concerning each of display content in a screen.

Step S2 in FIG. 5 is as explained above. In steps S12 and S22, only aviewer and content being provided are different. A viewing situation iscalculated or estimated as in the steps in FIG. 5.

[Processing Step S3: Content Evaluating Step]

FIG. 6 is a flowchart of the processing step S3. In the processing stepS3, the server Ts evaluates, on the basis of the calculated or estimatedviewing situations of the viewers, content in a time segment in whichthe viewers are likely to have viewed the contents at own wills of theviewers has high viewing importance. That is, the server Ts calculatesviewing importance on the basis of a viewing situation Good ratioindicating that a ratio of a viewing situation Good in which both of theviewing receptive level and the viewing controllable level of theviewing situation are the H level is high. The server Ts evaluatescontent in the time segment (the time slot) having the high viewingimportance to be content of high importance. Further, the server Ts mayevaluate the content importance taking into account provisionimportance, which is importance for the content provider side.

FIG. 23 is a diagram depicting an example of importance evaluation dataof content. In FIG. 23, a temporarily-created content viewing situationlist, importance label designation to be used, which is one ofregistered data, and content importance, which is evaluation data, aredepicted. Time slots in FIG. 23 may be fixed times or may be unfixedtimes. In this example, the times slots are fixed times. For example, atime slot: 1 of the content C1 is from a start 0 second of content to 1second, and a time slot: 2 is from the start 1 second to 2 second.Processing steps S301 to S304 in FIG. 6 are explained below withreference to the time slots.

S301: Viewing Situation Data Collecting Step

The server Ts collects data of viewing situations for the content C1 ofviewers in a group whose content is desired to be evaluated among theviewers including the viewer A (S301). For example, the server Tscollects data of viewing situations for the content C1 of viewers in asales group.

Specifically, the server Ts collects window and content displayhistories concerning the content ID: C1 on the basis of the datadepicted in FIG. 16. The displayed content history includes all recordsof all users. Therefore, when content evaluation is performed, whencontent is evaluated for a certain user group, it is preferable toevaluate the content using only data concerning the user group. Forexample, when evaluation target content is an explanation videoconcerning a product E, it is assumed that a point important for a userin charge of sales of the product E and a point important for a userplanning to purchase the product E are different. For example,concerning the content C1, there is already a past viewing log in anoffice where a large number of people in charge of sales are registered.The server Ts extracts only data collected on Jul. 12, 2012 depicted inFIG. 16 from a database to extract only viewing situation data of thepast viewing log data. As explained below, the server Ts calculatesimportance of content in the time slots from a viewing situation on Jul.12, 2012 of the content ID: 1.

S302: Viewing Situation Good Ratio Calculating Step

The server Ts calculates a ratio of a good viewing situation (S302). Asdepicted in FIG. 23, the good viewing situation (viewing situation Good)means that both of a viewing receptive level and a viewing controllablelevel are the H level in corresponding time slots in the collected dataon Jul. 12, 2012. A viewing situation Good ratio of a time slot 1 atthis point is calculated as percentage of viewing situation Good in thetime slot 1 with respect to a total number of viewing situation data ofthe time slot 1 of the collected data on Jul. 12, 2012.

S303: Viewing Importance Calculating Step

The server Ts calculates viewing importance on the basis of the viewingsituation Good ratio (S303). As depicted in FIG. 6, the viewingimportance is determined as Middle if the viewing situation Good ratiois between 40% and 60%, determined as Low if the viewing situation Goodratio is equal to or lower than 40%, and determined as high if theviewing situation Good ratio is equal to or higher than 60%.

S304: Content Importance Calculating Step

The server Ts calculates content importance on the basis of the viewingimportance (S304). Basically, the content importance is the same as theviewing importance. However, the server Ts may calculate the contentimportance partially taking into account provision importance. In thatcase, the server Ts determines the importance of the time slots of thecontent on the basis of the provision importance and the viewingimportance. For example, as depicted in FIG. 6, basically, the viewingimportance is the content importance. That is, the contentimportance=the viewing importance. However, concerning a time slot,provision importance of which is determined as the H level on theprovision side, even if viewing importance of a user group, which hasalready viewed content, set as a target this time is the Middle level,the server Ts evaluates content importance as the H level. In this way,the server Ts performs content evaluation setting a higher value on theviewing importance.

After the content evaluation, all temporarily created data depicted inFIG. 22 may be left. However, for which user group importance isevaluated may be managed as an importance label. The importance of thetime slots of the evaluated content and the importance label areassociated. Consequently, it is possible to manage importance evaluationdata of content for each of user groups.

The viewing importance may limit data used for calculation according totime, a user, a screen, and the like.

In FIG. 23, an example of registered data of the importance label to beused for each of the user groups is depicted. The content importance isrecorded for each of importance labels and for each of the time slots ofthe content C1.

FIG. 7 is a diagram depicting, as a specific example, an example inwhich content importance of the content C1 of the user A is evaluatedfor each of time slots. The lateral direction corresponds to 1 to N of atime slot tsl. Provision importance and viewing importance correspondingto the time slot tsl are depicted. Content importance evaluated on thebasis of the provision importance and the viewing importance isdepicted. In this specific example, the content importance is evaluatedas the H level when the time slot tsl is 3 to 10. In particular, whenthe time slot tsl is 10, the content importance is the H level becausethe viewing importance is Middle but the provision importance is Highlevel.

[Processing Steps S14 and S14-1: Feedback Desirable Content RegisteringStep]

FIG. 8 is a flowchart of the feedback desirable content registering stepS14-1. The feedback desirable content registering step S14-1 is aregistering step in the second and third content provision processing inFIGS. 3 and 4. The registering step S14 in the first content provisionprocessing in FIG. 1 is the same as the step S14-1 except that contentimportance is not taken into account.

FIG. 9 is a diagram for explaining the feedback desirable contentregistration processing. In FIG. 9, an example of (1) (2) contentimportance data stored in the content evaluation processing step S3, (3)a viewing situation of the content C1 of the present user D, and (4)feedback part distinguishing data are depicted.

In the feedback desirable content registering step S14-1, the server Tsregisters, on the basis of a viewing situation, as feedback desirablecontent FX, content in a time segment in which a viewer is likely tohave failed to view the content and in which content importance is high.That is, when the viewer is in a viewing situation in which viewing isuncontrollable in meeting with a guest or receiving a telephone call andthe viewer is determined as not being able to view content much, theserver Ts registers, as a point to be displayed as feedback in future,content displayed in such a time frame. Processing steps S1401 to S1407in FIG. 8 are explained below with reference to FIG. 9.

S1401

The server Ts extracts or checks a viewing situation concerning thescreen B of the user D at time t0 to time t1 stored in step S12 (S1401).When it is assumed that, for example, multiple content is displayed onthe screen B, a viewing situation of the content C1 among the multiplecontent is depicted in (3) of FIG. 9.

S1402

The server Ts determines whether a viewing receptive level of theviewing situation is the L level (S1402). In the case of L level, theserver Ts determines that the user D does not set the visual attentionon the screen. In this determination, the server Ts may determinewhether both of a viewing receptive level and a viewing controllablelevel are the L level. In the case of L/L level, the server Tsdetermines that the user D may be unable to view content because of aninevitable accident such as leaving seat for a telephone call.

S1403

When the viewing receptive level is the L level (YES in S1402), theserver Ts repeats processing explained below for all the contentdisplayed on the screen B to the user D.

S1404, S14-105, and S1406

For each time slot tsl of the content C1 at time t0 to time t1 (S1404),if content importance of the time slot tsl of the content C1 is the Hlevel (YES in S14-105), the server Ts sets a feedback necessary flag ofthe time slot tsl of the content C1 to true (S1406). The server Tsperforms this processing for all time slots of content.

S1407

The server Ts executes the processing S1404, S14-105, and S1406 on allthe content displayed on the screen B (S1407).

In the example depicted in FIG. 9, the evaluated content importance dataof the content C1 to the users A and the like is the H level in timeslot tsl=3 to 10 of the content C1. A viewing receptive level of thepresent viewing situation is the H level in the time slot tsl=3 to 10 ofthe present content C1. Therefore, in feedback part distinguishing data,the feedback necessary flag in the time slot tsl=3 to 10 is true.

FIG. 24 is a diagram depicting a registration example of feedbackcontent. In FIG. 24, the feedback necessary flag is true in a time slot3 to 5 of the content C1 displayed on the screen B of the user D. Thismatches FIG. 9.

In the first content provision processing in FIG. 2, content importanceis not taken into account in the feedback desirable content registrationprocessing S14. Therefore, in the first content provision processing,when the viewing receptive level is the L level, a time slot of contentat that point is registered as feedback desirable content.

[Processing Step S23: Composite Content Creating Step]

FIG. 10 is a flowchart of the composite content creating step S23. Inthe composite content creating step S23, when a time segment estimatedon the basis of a viewing situation as being in a viewable state comes,in a time segment in which content importance of content being providedis low, the server Ts combines the feedback desirable content FX withthe content being provided. That is, when a viewer is in a viewingsituation in which the viewer can be determined as concentratedlyviewing content without any interference, if there are feedbackdesirable content, in order to provide the feedback desirable content attiming when content importance of the content currently being vieweddecreases, the server Ts performs pre-display processing for the timing,combines the feedback desirable content with the content being provided,and prepares for display.

Alternatively, contrary to the above, the server Ts may combine thefeedback desirable content with a content part with low importance inadvance to prepare to be capable of displaying content and, afterchecking a viewing situation of a user, immediately perform presentationof feedback information right after it is determined that the viewingsituation has no problem for feedback information reception.

FIG. 11 is a diagram for explaining a composite content creating step.In FIG. 11, (1) a viewing situation of the content C1 or C2 for thepresent user D, (2) content importance of the content C1 or C2 beingprovided of the user D, (3) a feedback candidate time slot group, and(4) composite content for the content C1 or C2 being provided of theuser D are depicted. The flowchart of FIG. 10 is explained withreference to FIG. 11.

First, as a precondition, the server Ts records a viewing situation of aviewer calculated substantially on a real time basis. The server Tsacquires a recorded viewing situation concerning content currently beingprovided in a time segment of time t0 to time t1 (S2301) and startsprocessing for determining whether the viewer can receive feedbackdesirable content in the viewing situation and processing fordetermining whether feedback desirable content is present (S2302).Concerning the viewing situation, when both of a viewing receptive leveland a viewing controllable level of the screen B of the user D is the Hlevel, this means that the user D is in a good state of a viewingsituation for contents on the screen B. When the viewing situation isgood, if feedback desirable content is provided while being incorporatedin the content displayed on the screen B of the user D, it is highlylikely that the user D can also view the feedback desirable content in agood state.

Therefore, being triggered by the fact that both of the viewingreceptive level and the viewing controllable level are the H level, theserver Ts thereafter monitors, concerning the content currently beingviewed, when is timing with low content importance and performs editingof content beforehand such that the feedback desirable content can becombined and displayed in a time slot with low content importance.

In an example explained in this embodiment, feedback is performed atnearest timing with low content importance. However, a point forperforming feedback may be determined according to the H or L level ofone of viewing importance and provision importance. Alternatively, thepoint for performing feedback may be determined using not only dataallocated to content in advance such as the viewing importance and theprovision importance but also dynamic data such as a situation in whichit is possible to estimate at a high probability on the basis of asituation during viewing of the user D and a situation during viewing ofother users that a user determines that the importance of content is lowon the basis of, for example, similarity to a viewing situation and anattentive audience sensing data peculiar to time when content with lowcontent importance is viewed.

In the determination, the feedback desirable content to be combined andembedded and the content currently being displayed do not need to be thesame. The total length of time slots of the feedback desirable contentto be embedded and the total length of time slots of the contentcurrently being displayed do not need to be the same.

First, when content displayed on the screen B of the user D is two kindsof content, i.e., the content C1 and the content C2, the server Tschecks whether a time slot with low content importance is present in atime slot (t_(0+i)) to (t_(1+i)) (i=0−z) during from t0 until z secondsafter time t0 in any one of the two kinds of content (S2303, S2304, andS2305). The z seconds is set to, for example, time in which a good stateof a viewing situation can be expected to continue. Further, a sectionof the z seconds may be set after minimum time for the series ofcalculation or determination processing taking into account acalculation time. The z seconds is set to about 60 seconds.

When there is the time slot (t_(0+i)) to (t_(1+i)) with low contentimportance in z(=60) seconds from time t0 (YES in S2305), the server Tsrecords the time slot as a candidate of a time slot in which thefeedback desirable content are combined and embedded (S2306). If thecandidate is absent in the content C1, the server Ts searches for thecandidate in the remaining content C2 (S2308).

When the candidate is found in the content C1 or the content C2, theserver Ts specifies the position of a time slot continuing longest inthe content in which the candidate is found earlier (S2309). In theexample depicted in FIG. 11, the position of the continuing time slot istsl=20 to 27 of the content C1.

As depicted in FIG. 11, according to the flowchart of FIG. 10, when bothof the viewing receptive level and the viewing controllable level of theviewing situation concerning the screen B of the user D are the H levelat time t0 to time t1 and a time slot in which the feedback necessaryflag is true is present (YES in S2302), with that as a trigger, theserver Ts checks in advance, for all the displayed content (S2303),whether the importance of the content being viewed is the L level(S2305) concerning the time slot (t_(0+i)) to (t_(1+i)) (i=0−z) untilthe z time after time t0 (S2304). When the server Ts detects a time slotin which the content importance is the L level, the server Ts registersthe time slot tsl in a time slot group T of feedback candidates.

The server Ts leaves, in the time slot group T, a longest continuingtime slot tslc among time slots registered in the time slot group T(S2309). In the example depicted in FIG. 11, a time slot 20 to 27 is alongest continuing time slot. The server Ts performs content displaypre-processing for combining the feedback desirable content with thetime slot 20 to 27 of the content being provided (S2310).

FIG. 25 is a flowchart of the content display pre-processing S2310. Thecontent display pre-processing S2310 is processing for creatingcomposite content in FIG. 10. Processing of the flowchart is explainedbelow.

As depicted in FIGS. 10 and 11, the server Ts performs processing (S53and S54) for combining and embedding in advance the feedback desirablecontent in the detected time slot tsl=20 to 27 of the content C1 on thescreen B of the user D. First, the server Ts determines a screen area“a” appropriate for embedding the feedback desirable content in thecontent C1, which is content being provided (S50). The screen area “a”is desirably an area in which a quantity of information of the contentbeing provided is small. Therefore, the server Ts analyzes a frame imageof content included in the time slot tsl=20 to 27 and examines anddesignates a screen area in which a motion and an information amount aresmall.

The server Ts defines, as tslc (consecutive tsl), a time slot to whichthe time slot tsl in the time slot group T depicted in FIG. 11 isconsecutive (S51). The server Ts defines, as FC, a set of content fc(feedback desirable content fc) registered with the feedback necessaryflag=true, defines, as fc, content of elements of the set FC, defines,as ftsl, a time slot registered as the feedback necessary flag=true ofthe content fc, and defines, as FTSL, a set of the time slot ftsl (S51).Further, the server Ts defines, as ftslc (combined of ftsl), a time slotto which the time slot ftsl is coupled (S51).

Subsequently, the server Ts determines in what kind of form the feedbackdesirable content is embedded (without thinning frames or by thinningframes) (S52). The feedback desirable content is long extending over aplurality of time slots in some case or are short in other cases.Different kinds of content are sometimes included in the feedbackdesirable content.

Therefore, concerning all content and all time slots for which thefeedback necessary flag is set concerning the screen B of the user Ddepicted in FIG. 24, the server Ts calculates total length ftslc of atime slot for which feedback is needed. When the length tslc of the timeslot in the time slot tsl=20 to 27 of the content being provided islonger than the total time segment length ftslc of the time slot of thefeedback desirable content (YES in S52), the server Ts combines areduced size version of a frame image of the time slot of the feedbackdesirable content in the position of the area “a” of frame images in thetime slot tsl=20 to 27 of the content being provided. On the other hand,when the length tslc of the time slot in the time slot tsl=20 to 27 ofthe content being provided is shorter than the total time segment lengthftslc of the time slot of the feedback desirable content (NO in S52),the server Ts combines only a first part of the feedback desirablecontent to be embedded in the time slot tslc of the content beingprovided. Alternatively, as in the example depicted in FIG. 25, theserver Ts may create a main point slide show image from a video of allthe time slots of the feedback desirable content to be fit within thetime slot tsl=20 to 27 of the content being provided and combine themain point slide show image as a reduced image in the position of thearea “a” in the time slot tsl=20 to 27 of the content being provided(S54).

Finally, the server Ts updates a feedback reservation (S55).

When the composite content of the feedback desirable content is created,it may be necessary to perform switching control of a screen duringfeedback from the content being provided to the composite content andconversely switching the composite content to the content being providedwhile monitoring a time slot viewed by a user. As a simple method foravoiding the switching control, the feedback desirable content may bepresented in a new small window to overlap above a window of the contentbeing provided rather than being combined.

Further, in the above explanation, as a form of the feedback desirablecontent, it is selected according to the length of the time slot of thecontent being provided whether all videos of the time slot of thefeedback desirable content are presented or the main point slide show ispresented. However, the priority of presentation may be determined usingcontent importance to present only content with high priority or a formof presentation of the feedback desirable content may be determinedbeforehand for each of users, user groups, and content.

Concerning timing for performing feedback, in the example explained inthis embodiment, within a certain fixed time, feedback is reserved inthe longest coupled time slot tslc in which the content importance isthe L level. However, after a viewing situation of a user is checked, ifthe user can receive feedback information, the feedback desirablecontent may be immediately presented without taking into account theimportance of the content being currently viewed. It goes without sayingthat timing for presentation may be determined beforehand for each ofusers, user groups, and content.

FIG. 26 is a diagram depicting an example of feedback reservation data.In FIG. 25, content importance is low in the time slot tsl=20 to 27 ofthe content C1 displayed on the screen B of the user D. Content in astate in which the time slot ftsl=3 to 10 of the content C1 marked as afeedback point is embedded is already created. As shown in FIG. 26, thefeedback reservation data is depicted as the user D, the screen B, thefeedback desirable content C1, the time slot ftsl=3 to 10, the compositecontent C1 being presented, and the time slot tsl=20 to 27.

These data are temporary content. Therefore, the data are not includedin the configuration diagram of the server Ts depicted in FIG. 14. As aform of stored data, a form like a content table depicted in FIG. 16 isa basic form. A column for associating the table and a table depicted inFIG. 26 only has to be present in any one of the tables.

[Processing Step S24-2: Feedback Desirable Content Provision ProcessingStep]

FIG. 12 is a flowchart of the feedback desirable content provisionprocessing step S24-2. In the feedback desirable content presentationprocessing step S24-2, the server Ts presents composite content embeddedwith the feedback desirable content in a time segment in which it isestimated on the basis of a viewing situation that a user is in aviewable state and in which content importance of content being providedis the L level.

That is, when the user is in a viewing situation in which it is possibleto determine that the user is concentratedly viewing content without anyinterference, when there is a feedback reservation not executed yet, ifthe content currently being viewed enters a time segment in whichfeedback is reserved, the server Ts displays an image of the compositecontent reserved in FIG. 26.

FIG. 13 is a diagram for explaining the feedback desirable contentprovision processing step. In FIG. 13, (1) a viewing situation of thecontent C1 or C2 for the present viewer D, (2) a display situation of acomposite content of the content C1 or C2 being provided of the viewerD, and (3) feedback part distinguishing data in which the feedbacknecessary flag is recorded are depicted. The flowchart of FIG. 12 isexplained with reference to FIG. 13.

After calculating or estimating and recording a user viewing situationcalculated or estimated substantially on a real time basis, the serverTs acquires the stored viewing situation (S2401) and starts, concerningcontent in a time segment of recorded time t0 to time t1, determinationprocessing for determining whether a viewer is in a viewing situation inwhich the viewer can receive the feedback desirable content anddetermination processing for determining whether there is a feedbackreservation (S2402). When both of the viewing receptive level and theviewing controllable level of the screen B of the user D are the Hlevel, this means that the user D is in a good state of a viewingsituation for content on the screen B. In such a good state of theviewing situation, the server Ts determines whether there are feedbackdesirable content for which a feedback reservation is made but feedbackis not executed yet according to whether there are feedback desirablecontent in which a feedback reservation execution flag is false (S2402).

When there is data for which feedback is not executed in the good stateof the viewing situation (YES in S2402), the server Ts checks,concerning a composite content corresponding to content being currentlyprovided, whether a time slot of the content being currently providedcoincides with the time slot tsl=20 to 27 of reserved content (S2403).That is, when the next time slot (tsl=present time slot+1) of thepresent time slot is within a time slot between a reservation start timeslot tsl=20 and a reservation end time slot tsl=27 (S2403), the serverTs performs display of the composite content (S2404).

A display form of the feedback desirable content is not limited to thecomposite content and may be a form including a user interface foracquiring information concerning the combine content. In that case, theviewer can perform operation to display the composite content on thebasis of the user interface.

Subsequently, the server Ts changes the feedback reservation executionflag of the content C1 on the screen B of the user D to true and setsthe feedback reservation to an executed state (S2405). If the reservedtime slot tsl=20 to 27 elapses in the content C1, the server Ts switchesthe content being provided to the original content (S2407).

The server Ts stores the display of the composite content as a displayhistory as depicted in FIG. 16. In storing the display of the compositecontent, concerning the displayed time slot, the server Ts rewrites thefeedback necessary flag in the feedback part distinguishing data in FIG.24 to false and stores that feedback is already performed (S2408). InFIG. 13, it is depicted that (3) the feedback necessary flag of thefeedback part distinguishing data is rewritten from true to false.

FIG. 27 is a flowchart of the processing step S2408. The processing stepS2408 is explained with reference to FIG. 13 as well. First, concerninga time slot of the content being provided, when a reservation start timeslot of feedback composite content is represented as ts (=20), areservation end time slot is represented as te (=27), and the number ofdisplayed time slots is represented as k, a ratio p of feedbackcompletion is represented as p=k/(te−ts). On the other hand, when afeedback point start time slot of the composite content is representedas fts (=3) and an end time slot is represented as fts (=10),feedback-completed final time slot ftv is represented asftv=fts+(fte−fts)*p. That is, the feedback-completed final time slot ftvin the composite content is calculated at a ratio same as the feedbackcompletion ratio p in the content being provided.

The server Ts changes the feedback necessary flag in the feedback partdistinguishing in FIG. 24 to false, which indicates feedback completion,from the time slot fts (=3) to the time slot ftv (=any one of 3 to 10).When a feedback reservation is left without being executed for a fixedperiod, the server Ts may perform processing for automaticallycancelling the reservation.

As explained above, according to this embodiment, even when an emotionalresponse is less easily obtained from a viewer as in E-learning, it ispossible to collect attentive audience sensing data for quantity ofinformation contact measurement and acquire a viewing situation of theviewer. It is possible to evaluate the importance of certain content onthe basis of the viewing situation of the viewer. It is possible tospecify, on the basis of the viewing situation of the viewer, contentthat the viewer is likely to have failed to view and register thecontent as feedback desirable content. Further, it is possible toprovide, on the basis of the viewing situation of the viewer, thefeedback desirable content under a situation in which the viewer canview content.

In the embodiment, the viewing of a video of a speech or E-learning isexplained on the basis of a situation in which a viewer views the videoin front of a desktop screen. However, the viewing is not limited tothis. What a user is viewing may be visualization of a real world ratherthan a delivered video.

With a head mounted display (HMD) with a camera, a real world around auser viewed by the user can be sampled by the camera. In particular,when the user acts alone, a viewing reaction of the user alone is lesseasily depicted. It is difficult to determine the importance of aviewing target from a facial expression, a shout of joy, and the like.Therefore, it is possible to more appropriately determine the importanceof viewing target content by comprehensively evaluating viewingsituations of a large number of users. In that case, the length and theorder of time slots are different depending on the users. It is possibleto calculate a viewing Good ratio of the time slots by performingprocessing such as association of similar time slots.

All examples and conditional language provided herein are intended forthe pedagogical purposes of aiding the reader in understanding theinvention and the concepts contributed by the inventor to further theart, and are not to be construed as limitations to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although one or more embodiments of thepresent invention have been described in detail, it should be understoodthat the various changes, substitutions, and alterations could be madehereto without departing from the spirit and scope of the invention.

What is claimed is:
 1. A non-transitory computer readable storage mediumstoring a content providing program causing a computer to executecontent provision processing comprising: acquiring a first viewingsituation of a first viewer viewing a provided content in associationwith time when the content is provided; acquiring second viewingsituations of a plurality of second viewers viewing the provided contentin association with the time when the content is provided; evaluating,on the basis of the acquired second viewing situations of the pluralityof second viewers, content in a time segment, in which the plurality ofsecond viewers are likely to have viewed the content at own wills of thesecond viewers, to be content of high importance; registering feedbackdesirable content in a time segment being overlapped by a first timesegment in which the first viewer is likely to have failed to view thecontent on the basis of the first viewing situation of the first viewer,and a second time segment in which the plurality of second viewers arelikely to have viewed the content at own wills of the second viewers onthe basis of the acquired second viewing situations of the plurality ofsecond viewers; estimating, on the basis of the first viewing situationof the first viewer, a state in which the first viewer is enabled toview the content; and providing the first viewer with the feedbackdesirable content in a time segment in which estimation is made that thefirst viewer is in a viewable state.
 2. The non-transitory computerreadable storage medium storing the content providing program accordingto claim 1, wherein, in the provision of the feedback desirable content,the feedback desirable content is provided to the first viewer in a timesegment in which estimation is made that the first viewer is in theviewable state and also in which the content importance is low.
 3. Thenon-transitory computer readable storage medium storing the contentproviding program according to claim 1, wherein the content provisionprocessing further comprises generating a composite content bycombining, in the time segment where estimation is made that the firstviewer is in the viewable state, the feedback desirable content withcontent being provided in a time segment in which the content beingprovided is content of low importance, and in the provision of thefeedback desirable content, the composite content is provided to thefirst viewer in the time segment in which estimation is made that thefirst viewer is in the viewable state.
 4. The non-transitory computerreadable storage medium storing the content providing program accordingto claim 1, wherein the viewing situation includes a viewing receptivelevel indicating at what degree the viewer is enabled to view thecontent at own wills of the viewer and also indication a viewingcontrollable level indicating at what degree the viewer is enable tocontrol the viewing situation of the viewer viewing the content, in theacquisition of the first viewing situation of the first viewer oracquisition of the second viewing situations of the second viewers, inaccordance with a length of time in which an actual visual attention ofthe viewer stays in a content display region of a display screen duringthe provision of the content, the viewing receptive level is evaluatedto be higher as the staying time is longer, and, in accordance to anevent for increasing or reducing an amount of information contact withthe content during the provision of the content, the viewingcontrollable level is evaluated to be higher as the amount ofinformation contact increases, and in the evaluation of the contentimportance, a time segment, in which both of the viewing receptive leveland the viewing controllable level are high, is evaluated as a timesegment, in which the plurality of second viewers are likely to haveviewed the content at own wills of the second viewers.
 5. Thenon-transitory computer readable storage medium storing the contentproviding program according to claim 4, wherein, in the registration ofthe missed content as the feedback desirable content, a time segment inwhich the viewing receptive level is low is evaluated as a time when thefirst viewer is likely to have failed to view the content.
 6. Thenon-transitory computer readable storage medium storing the contentproviding program according to claim 4, wherein, in the providing thefirst viewer with the feedback desirable content, the time segment, inwhich both of the viewing receptive level and the viewing controllablelevel are high, is estimated as a time segment, in which the firstviewer is in a viewable state.
 7. The non-transitory computer readablestorage medium storing the content providing program according to claim3, wherein, the viewing situation includes a viewing receptive levelindicating at what degree the viewer is enabled to view the content atown wills of the viewer and also indication a viewing controllable levelindicating at what degree the viewer is enable to control the viewingsituation of the viewer viewing the content, in the acquisition of thefirst viewing situation of the first viewer or acquisition of the secondviewing situations of the second viewers, in accordance with a length oftime in which an actual visual attention of the viewer stays in acontent display region of a display screen during the provision of thecontent, the viewing receptive level is evaluated to be higher as thestaying time is longer, and, in accordance to an event for increasing orreducing an amount of information contact with the content during theprovision of the content, the viewing controllable level is evaluated tobe higher as the amount of information contact increases, and in thegeneration of composite content, the time segment, in which both of theviewing receptive level and the viewing controllable level are high, isevaluated as the time segment, in which estimation is made that thefirst viewer is in the viewable state.
 8. A content providing method forcausing a computer to execute content provision processing for providinga viewer with content, the content provision processing comprising:acquiring a first viewing situation of a first viewer viewing a providedcontent in association with time when the content is provided; acquiringsecond viewing situations of a plurality of second viewers viewing theprovided content in association with the time when the content isprovided; evaluating, on the basis of the acquired second viewingsituations of the plurality of second viewers, content in a timesegment, in which the plurality of second viewers are likely to haveviewed the content at own wills of the second viewers, to be content ofhigh importance; registering feedback desirable content in a timesegment being overlapped by a first time segment in which the firstviewer is likely to have failed to view the content on the basis of thefirst viewing situation of the first viewer, and a second time segmentin which the plurality of second viewers are likely to have viewed thecontent at own wills of the second viewers on the basis of the acquiredsecond viewing situations of the plurality of second viewers;estimating, on the basis of the first viewing situation of the firstviewer, a state in which the first viewer is enabled to view thecontent; and providing the first viewer with the feedback desirablecontent in a time segment in which estimation is made that the firstviewer is in a viewable state.
 9. A content providing apparatus thatprovides a viewer with content, the content providing apparatuscomprising: a first viewing situation acquiring unit configured toacquire a first viewing situation of a first viewer viewing a providedcontent in association with time when the content is provided; a secondviewing situation acquiring unit configured to acquire second viewingsituations of a plurality of second viewers viewing the provided contentin association with the time when the content is provided; an evaluatingunit configured to evaluate, on the basis of the acquired second viewingsituations of the plurality of second viewers, content in a timesegment, in which the plurality of second viewers are likely to haveviewed the content at own wills of the second viewers, to be content ofhigh importance; a feedback desirable content registering unitconfigured to register feedback desirable content in a time segmentbeing overlapped by a first time segment in which the first viewer islikely to have failed to view the content on the basis of the firstviewing situation of the first viewer, and a second time segment inwhich the plurality of second viewers are likely to have viewed thecontent at own wills of the second viewers on the basis of the acquiredsecond viewing situations of the plurality of second viewers a viewablestate estimating unit configured to estimate, on the basis of the firstviewing situation of the first viewer, a state in which the first vieweris enabled to view the content; and a feedback desirable contentproviding unit configured to provide the first viewer with the feedbackdesirable content in a time segment in which estimation is made that thefirst viewer is in a viewable state.